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Middleware Trade Study for NASA Domain 


Abstract 

This presentation presents preliminary results of a trade study designed to 
assess three distributed simulation middleware technologies for support of 
the NASA constellation Distributed space Exploration Simulation (DSES) 
project and Test and verification Distributed System integration Laboratory 
(DSIL). The technologies are: the High Level Architecture (HLA), the Test and 
Training Enabling Architecture (TENA), and an XML-based variant of 
Distributed interactive simulation (Dis-XML) coupled with the Extensible 
Messaging and Presence Protocol (XMPP). According to the criteria and 
weights determined in this study, hla scores better than the other two for 
dses as well as the dsil. 
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'Zack Crues, JSC, DSES Lead - Study POC 
•David Hasan, L3 Communications - Study Lead 
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•Danny Thomas, Aegis Technologies 
•Bobby Hartway, Aegis Technologies 
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Study Purpose and Scope 
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•Study Purpose 

- Answer the question: 

•Which of three candidate middleware technologies is best in 
Distributed Simulation Exploration Simulation (DSES) and 
Distributed System Integration Laboratory (DSIL)? 

-High Level Architecture (HLA) 

-Test and Training Enabling Architecture (TENA) 

-XML-based version of the Distributed Interactive Simulation (DIS) 
(using Extensible Messaging and Presence Protocol (XMPP) as 
messaging protocol) 

•Study Scope 

- Evaluated relative merits of the candidates against each other 

- Did not address: 

•General architecture questions (e.g., for DSIL, geographical 
distribution of time sensitive components) 

•Other (e.g., custom development of a distributed middleware) 


Candidate Descriptions 


•HLA 

- Originated with DoD as a standard set of services for linking distributed simulations and 
training applications; now IEEE standard (1516) with commercially available Run-Time 
Infrastructure (RTI) implementations 

- Does not specify on-the-wire data representations 

- Specifies a set of rules that “federates” must obey to form a “federation” and set of services 
(with C++ and Java mappings) through which the federate simulations interact with each 
other and the RTI 

•TENA 

- Originated with DoD; designed to support interoperability and reuse among DoD test and 
training ranges 

- Provides object-oriented approach for real-time exchange of data and invocation of 
remotely located objects 

- DoD Central Test and Evaluation Program (CTEIP) sponsors TENA middleware 
development and distributes the only implementation 

• DIS-XML/XMPP 

- Originated with DoD; defines on-the-wire protocol now adopted as IEEE standard 1278 

- DIS-XML utilizes Extensible Markup Language (XML) to encode DIS data on the wire to 
take advantage of wide availability of XML-processing tools and standardization 

- Jabber/XMPP chat room concept (though not explicitly intended for distributed simulation) 
can be effectively used as a communications mechanism for distributed simulations 
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Summary 


This briefing provides a summary of NASA Constellation DSES/DSIL 
Distributed Simulation Middleware Trade Study, June 2007. 

•Conclusions 

- DIS-XML/XMPP falls far short of what we need for DSES and DSIL, 

- HLA is the best solution for DSES and 

- Even in the DSIL, HLA comes out ahead. 

•Caveats 

-Criteria weights and some raw scores derived from subjective judgments 
of the Study Team 

• Data and weights are available for review 

- Plan to complete benchmark codes as standardized test suite 

* Results of the study not sufficiently sensitive to latency and throughput 
scores for the benchmark results to affect conclusions 


Method 


•Scored the three middleware technologies against a set of more than 20 
technical criteria and multiplied the scores by DSES and DSIL-specific 
weights to derive an overall grade for each technology as applied in each 
context (DSES and DSIL) 

•Assignment of DSES and DSIL-specific weights to each criterion (eg, non 
real-time capabilities important for DSES, not as important for DSIL) 

• “Raw” scores developed relative to each criterion for each of the three 
technologies 

- Drawn from a “pool” of 100 points for each criterion and distributed among the three 
technologies relative to how well each performs relative to the others 

- Some raw scores based on quantitative data (e.g., latency); others based on presence 
or absence of certain capabilities (e.g., synchronization); others based on team 
consensus of the relative strengths and weaknesses of the candidates 

•Weighted grades for each technology, for each application were 
developed for each criterion, along with on overall score for each 
technology for each application 


Evaluation Criteria Categories 


Candidates evaluated against 26 criteria in the following 
categories: 

•User operations 
•Time response 
•Architectural robustness 
•Performance 

•Efficient resource utilization 


Evaluation Criteria - User Operations 
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•Synchronization - Ability to facilitate a coordinated, consistent 
initialization of common simulation parameters and maintain causality 
(accurate representation of cause and effect/ stimulus and response 
relationships among the set of executing interoperable simulations) 

•Compile time data checks - Ability to detect data type inconsistencies 
early on during development instead of later during simulation testing. 

•Save and restore - Ability to save the state of a simulation, and at a 
later time restart this simulation from that time with identical states. 

•Data reduction/analysis - Tools/capabilities/features for data 
reduction/analysis/reporting. 

•Data viewers - Tools/capabilities/features for visualization of run results 
and run replays. 

•Flexible data exchange - Flexibility to allow system-to-system data 
transfer using Cx-defined standards/protocols (e.g., via C3I 
specification) or other data exchange mechanism or protocols. 

•Data recording - Run-time tools/capabilities/features for non-intrusive 
data recording (and playback) of data. 

•Data filters - Provide mechanisms to selectively distribute data. 


Evaluation Criteria - Time Response 


•Latency - Latency artifacts associated with application data 
exchange introduced by the middleware. 

•Throughput - Artifacts introduced by middleware that limits 
bandwidth supported by applications 

•Multiple concurrent executions - Provide for multiple, 
concurrent simulation executions over the same LAN/WAN 
communications network 

•Simulation time management - Ability to coordinate 
advancement of logical time (and its relationship to real-time) 
among simulation federates. Provides ability to support time 
coordination among both real-time and non real-time simulation 
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Evaluation Criteria - Architectural Robustness 


* 


•Recover from middleware crashes - Graceful recovery from 
middleware software faults. 

•Recover from network faults - Graceful recovery from network 
faults. 

•Recover from simulation crashes - Graceful recovery from 
simulation software faults. 
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Evaluation Criteria - Performance 


• Hardware-in-the-loop - Provide for integration / interoperation of HWIL/SWIL 
system representations, in addition to all digital simulation representations 

•Real-time operations - Provide for integrated operations in real-time (within 
limits of HW, SW, and OS) 

• Best effort delivery - Support both guaranteed delivery (via TCP/IP) and best 
effort (via UDP) message protocols." 

•Causality and repeatability - Obtain same results from one simulation run to 
the next with identical inputs. Causal implies that simulation events are in the 
same order they would occur in the real world, and that everybody sees events 
in the same order. 

• Distribution transparency - Ability to implement Cx-level simulations within 
physical proximity (eg, same Lab) or optionally distribute at various sites with 
minimal reconfiguration required 

• Dynamic conceptual models - Ability to transfer ownership of simulation object 
dynamics from one application to another during the simulation execution. 

• Multi-media support - Ability to support transfer of simulation video and voice 
during execution. 


Evaluation Criteria - Efficient Resource Utilization 


•CPU - CPU utilization requirements of the middleware. 

•Memory - CPU memory utilization requirements of the 
middleware. 

•Scalability and extensibility - The characteristic to readily scale 
in terms of added simulation objects and additional federates. 

•Execution startup time - Simulation initialization time. 
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Criteria Weighting 




DSES 

weights 

DSIL 

weights 


Performance 

OBJECTIVES 

Technical Performance Criteria 




1.1 Support User 
OPERATIONS 

1.1.1 Provide synchronization 

10.0% 

7.0% 

1.1.2 Provide compile time data checks 

2.0% 

2.0% 

1.1.3 Provide save & restore 

10.0% 

10.0% 

1.1.4 Provide data reduction/analysis tools 

1.0% 

1.0% 

1.1.5 Provide Data viewers 

1.0% 

1.0% 

1.1.6 Provide flexible data exchange 

1.0% 

5.0% 

1.1.7 Provide data recording tools 

3.0% 

4.0% 

1.1.8 Provide data filters 

1.0% 

1.0% 

1.2 Optimize Time 
RESPONSE of 
Architecture 

1.2.1 Minimize latency 

4.0% 

11.0% 

1.2.2 Optimize message throughput 

4.0% 

6.0% 

1.2.3 Support multiple, concurrent executions 

2.0% 

2.0% 

1.2.4 Support time management 

10.0% 

2.0% 

1.3 Maximize 

Architecture 

ROBUSTNESS 

1.3.1 Gracefully recover from middleware crashes 

2.0% 

2.0% 

1.3.2 Gracefully recover from network faults 

1.0% 

1.0% 

1.3.3 Gracefully recover from simulation crashes 

2.0% 

2.0% 

1 .4 Provide 
Required User 
PERFORMANCE 

1.4.1 Support HWIL 

8.0% 

13.0% 

1.4.2 Support Real-time M&S/operations 

8.0% 

13.0% 

1.4.3 Best Effort Delivery - TCP/UDP 

8.0% 

2.0% 

1.4.4 Support causal and repeatable M&S 

5.0% 

2.0% 

1.4.5 Provide distribution transparency 

4 . 0 - ■ 

2.0% 

1.4.6 Support dynamic conceptual models 

6.0% 

4.0% 

1.4.7 Provide mutli-media support services 

2.0% 

2.0% 

1 .5 Optimize 
Architecture 
Resource 
EFFICIENCY 

1.5.1 Optimize Node CPU Utilization 

1.0% 

1.0% 

1.5.2 Optimize Node CPU Memory Utilization 

1.0% 

1.0% 

1.5.3 Maximize scalability & extensibility 

2.0% 

2.0% 

1.5.4 Miminize Architecture (federate) start-up time 

1.0% 

1.0% 


| 100% | 100% I 


Criteria weights developed 
by Team’s judgment of different 
degrees of relevance of each 
criteria to the DSES and DSIL 
communities 

Scoring spreadsheet available 
to assess sensitivities to 
different weights 
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Latency Benchmarks 


Benchmark code was 
implementation of 
simple publish and subscribe 


•Data in milliseconds 
•Reliable message delivery 
•1000 samples/payload size 
•“1 way” 


TENA 

Size 

Average 

Std Dev 

Min 

Max 

1 

0.88 

0.22 

0.5 

1.5 

4 

0.62 

0.22 

0.5 

2 

16 

0.6 

0.3 

0.5 

6 

64 

0.6 

0.29 

0 

5.5 

256 

0.61 

0.27 

0.5 

5 

1024 

0.66 

0.28 

0.5 

4.5 

4096 

0.73 

0.31 

0.5 

5 


Single 4 CPU 
Machine at JSC 

HLA 

Size 

Average 

Std Dev 

Min 

Max 

1 

1.05 

0.76 

0.58 

20.58 

4 

1.11 

0.76 

0.59 

20.18 

16 

1.24 

0.78 

0.59 

20.82 

64 

1.39 

0.76 

0.59 

20.11 

256 

1.46 

0.7 

0.59 

20.64 

1024 

1.22 

0.74 

0.62 

21.29 

4096 

1.38 

0.68 

0.61 

18.33 


TENA 

1 

26.16 

15.55 

15.5 

361 

64 

16.9 

12.94 

15.5 

252.5 

4096 

47.17 

11.29 

32.5 

252.5 


JSC - MSFC 

HLA 

1 

15.77 

0.84 

14.62 

32.08 

64 

15.77 

0.79 

14.74 

31.49 

4096 

17.5 

9.04 

15.89 

171.12 


We did not find significant differences in latency 
performance between TENA and HLA 
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Raw Scores 


Performance 

OBJECTIVES 

Technical Evaluation Criteria 




I 


1.1 Support User 
OPERATIONS 

1.1.1 Provide synchronization 

1.1.2 Provide compile time data checks 

1.1.3 Provide save & restore 

1.1.4 Provide data reduction/analysis tools 

1.1.5 Provide Data viewers 

1.1.6 Provide flexible data exchange 

1.1.7 Provide data recording tools 

1.1.8 Provide data filters 

1.2 Optimize Time 
RESPONSE of 
Architecture 

1.2.1 Minimize latency 

1.2.2 Optimize message throughput 

1.2.3 Support multiple, concurrent executions 

1.2.4 Support time management 

1.3 Maximize Architecture 
ROBUSTNESS 

1.3.1 Gracefully recover from middleware crashes 

1 .3.2 Gracefully recover from network faults 

1 .3.3 Gracefully recover from simulation crashes 

1.4 Provide Required User 
PERFORMANCE 

1.4.1 Support HWIL 

1.4.2 Support Real-time M&S/operations 

1 .4.3 Best Effort Delivery - TCP/UDP 

1.4.4 Support causal and repeatable M&S 

1 .4.5 Provide distribution transparency 

1 .4.6 Support dynamic conceptual models 

1 .4.7 Provide mutli-media support services 

1.5 Optimize Architecture 
Resource EFFICIENCY 

1.5.1 Optimize Node CPU Utilization 

1.5.2 Optimize Node CPU Memory Utilization 

1 .5.3 Maximize scalability & extensibility 

1 .5.4 Miminize Architecture (federate) start-up time 


Raw Scores 


HLA TENA DIS- 

XMUXMP 


100.0 

0.0 

0.0 

10.0 

80.0 

10.0 

100.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

33.0 

33.0 

33.0 

0.0 

0.0 

0.0 

80.0 

20.0 

0.0 

33.0 

33.0 

33.0 

33.0 

33.0 

33.0 

33.0 

33.0 

33.0 

100.0 

0.0 

0.0 

50.0 

50.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

45.0 

45.0 

10.0 

45.0 

45.0 

10.0 

60.0 

40.0 

0.0 

100.0 

0.0 

0.0 

33.0 

33.0 

33.0 

100.0 

0.0 

0.0 

0.0 

100.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

40.0 

40.0 

20.0 

0.0 

0.0 

0.0 
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Weighted Grades 


(Assessment Score) 


Raw Scores 


OBJECTIVES 

Technical Evaluation Criteria 




HLA TENA DIS- 

XML/XMP 


100.0 

0.0 

0.0 

10.0 

80.0 

10.0 

100.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

33.0 

33.0 

33.0 

0.0 

0.0 

0.0 

80.0 

20.0 

0.0 

33.0 

33.0 

33.0 

33.0 

33.0 

33.0 

33.0 

33.0 

33.0 

100.0 

0.0 

0.0 

50.0 

50.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

45.0 

45.0 

10.0 

45.0 

45.0 

10.0 

60.0 

40.0 

0.0 

100.0 

0.0 

0.0 

33.0 

33.0 

33.0 

100.0 

0.0 

0.0 

0.0 

100.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

40.0 

40.0 

20.0 

0.0 

0.0 

0.0 


1.1 Support User 
OPERATIONS 

1.1.1 Provide synchronization 

1 .1 .2 Provide compile time data checks 

1.1 .3 Provide save & restore 

1 .1 .4 Provide data reduction/analysis tools 

1.1 .5 Provide Data viewers 

1 .1 .6 Provide flexible data exchange 

1 .1 .7 Provide data record ng tools 

1.1 .8 Provide data filters 

1.2 Optimize Time 
RESPONSE of 
Architecture 

15.1 Minimize latency 

1 2.2 Optimize message throughput 

12.3 Support multiple, concurrent executions £ 

1 24 Support time management 

1.3 Maximize Architecture 
ROBUSTNESS 

1.3 1 Gracefully recover from middleware crashes 

1 .3.2 Gracefully recover from network faults 

1 .3.3 Gracefully recover from simulation crashes , 

1 4 Provtie Required User 
PERFORMANCE 

1.4.1 Support HWIL 

1.4.2 Support Real-time M&S/operations i 

1.4.3 Best Effort Delivery -TCP/UDP 

1 .4.4 Support causal and repeatable M&S 

1 .4.5 Provide distribution transparency 

1 .4.6 Support dynamic conceptual models 

1 .4.7 Provide mutli-media support services 

1 .5 Optimize Architecture 
Resource EFFICIENCY 

1 .5.1 Optimize Node CPU Utilization 

1 .5.2 Optimize Node CPU Memory Utilization 

1 .5.3 Maximize scalability & extensibility 

1.5.4 Miminize Archtitecture (federate) start-up time 


(SE&I 
X Weight) 


DSES 

weights 


10.0% 

2.0% 

10.0% 

1.0% 

1.0% 

1.0% 

3.0% 

1.0% 

4.0% 

4.0% 

2.0% 

10.0% 

2.0% 

1.0% 

2.0% 

8.0% 

8 . 0 % 

8.0% 

5.0% 

4.0% 

6 . 0 % 

2.0% 

1.0% 

1.0% 

2.0% 

1.0% 



( 100 %) 


(SE&I-Weighted 

GRADE) 



10.00 

0.00 

0.00 

0.20 

1.60 

0.20 

10.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.33 

0.33 

0.33 

0.00 

0.00 

0.00 

0.80 

0.20 

0.00 

1.32 

1.32 

1.32 

1.32 

1.32 

1.32 

0.66 

0.66 

0.66 

10.00 

0.00 

0.00 

1.00 

1.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

3.60 

3.60 

0.80 

3.60 

3.60 

0.80 

4.80 

3.20 

0.00 

5.00 

0.00 

0.00 

1.32 

1.32 

1.32 

6.00 

0.00 

0.00 

0.00 

2.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.80 

0.80 

0.40 

0.00 

0.00 

0.00 


“ DSES - 
Subgrades 


nn 

1 2 1 1 | 



DIS 


RESPONSE grade 

nr 


SPONSE qr; 

a 


3| 


ROBUSTNESS grade 

I 1 I 1 I 0 I 


HLA I TENA I OtS 


PERFORMANCE 



EFFICIENCY grade 


mrn 


HLA | 

| TENA | 

| DIS 


(T&V 
X Weight) 


DSIL 

weights 


7.0% 

2.0% 

10.0% 

1.0% 

1.0% 

5.0% 

4.0% 

1.0% 

11.0% 

6.0% 

2.0% 

2.0% 

2.0% 

1.0% 

2.0% 

13.0% 

13.0% 

2.0% 

2.0% 

2.0% 

4.0% 

2.0% 

1.0% 

1.0% 

2.0% 

1.0% 


(SE&I-Weighted 

GRADE) 


DSIL grades 


HLA TENA DIS- 
XML'XMP 


7.00 

0.00 

0.00 

0.20 

1.60 

0.20 

10.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

1.65 

1.65 

1.65 

0.00 

0.00 

0.00 

0.80 

0.20 

0.00 

3.63 

3.63 

3.63 

1.98 

1.98 

1.98 

0.66 

0.66 

0.66 

2.00 

0.00 

0.00 

1.00 

1.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

5.85 

5.85 

130 

5.85 

5.85 

1.30 

1.20 

0.80 

0.00 

2.00 

0.00 

0.00 

0.66 

0.66 

0.66 

4.00 

0.00 

0.00 

0.00 

2.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.80 

0.80 

0.40 

0.00 

0.00 

0.00 


dsII 

Sub grades 

OPER ATIONS grade 

I 20 | 3 | 2 | 


HLA I TENA I OIS 












































Other Considerations 


* 


•Timing sensitivity in the DSIL - 

- Timing and latency issues will be drivers in the DSIL, but these are 
likely to present fundamental simulation design challenges that cannot 
be solved simply by selecting a middleware technology 

•Technology Maturity- 

- HLA is the more mature technology of the three we considered. 

•Vendor Independence - 

- Both HLA and TENA suffer from a kind of vendor dependence problem. 

• HLA implementations from different vendors do not interoperate. 

• For TENA, 

- Vendor independence problem derives from the fact that there is only a single 
implementation. 

- Furthermore, some aspects of TENA development (e.g., mapping an abstract 
object model into C++ code) must be done by the TENA office, possibly making 
them a critical link in the development cycle. 

•Middleware migration costs - 

- Were we to choose some other technology than HLA as the common 
middleware, we would likely have to justify that choice against the 
redevelopment costs 
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